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Abstract 

We show that the counting sequence for permutations avoiding both of the (clas- 
sical) patterns 1243 and 2134 has the algebraic generating function supplied by 
Vaclav Kotesovec for sequence A164651 in The On-Line Encyclopedia of Integer 
Sequences. 

1 Introduction 

Several authors have developed methods to count permutations avoiding a given set of 
patterns; see, for example, the references in the Wikipedia entry [']. In particular, enu- 
meration schemes have been developed for automated counting [ , ]. When successful, 
an automated method produces an enumeration scheme that yields the initial terms of 
the counting sequence (perhaps 20 or more) and sometimes these terms appear to have an 
algebraic generating function that does not follow readily from the enumeration scheme. 
Here, we treat one such case. The counting sequence for permutations that avoid both of 
the (classical) patterns 1243 and 2134 begins 1, 2, 6, 22, 87, 354, . . . , sequence A164651 
in The On-Line Encyclopedia of Integer Sequences [4]. In a comment on this sequence 
dated Oct 24 2012, Vaclav Kotesovec observed that the generating function 

3x 2 - 9x + 2 + x(l - x)y/l - Ax 
2(x -l)(x 2 + Ax- 1) 

fits the known terms of the sequence. 

We will show that {1243, 2134}-avoiders do indeed have this generating function. 
Defining a start-small permutation to be one that does not start with its largest entry, 
the proof rests on a bijection <ft from start-small {1243, 2134}-avoiders of length n to lists 
of start-small 123-avoiders whose total length is n — 1 + the length of the list. Under 
this bijection 0, for example, 12345 — > (12, 12, 12, 12), a list of length 4 with total length 
of its entries = 8, and 3412 — >■ (3412), a singleton list. Now, 123-avoiders are famously 
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counted by the Catalan numbers and the well known combinatorial interpretation of the 
transform (a n ) n >i — > (&„)„>i, defined via generating functions by 

permits counting these lists of start-small 123-avoiders. The generating function for start- 
small {1243, 2134}-avoiders thus obtained readily yields the generating function for un- 
restricted {1243, 2134}-avoiders. 

In Section 2, we show that is given by iteration of a more basic bijection, which 
is presented in Section 3. Finally, Section 4 gives the bookkeeping details to obtain the 
desired generating function from <fi. 



2 Reduction of problem 

A mid-123 entry in a permutation is an entry that serves as the "2" in a 123 pattern. 
A key mid-123 entry is a mid-123 entry b whose immediate predecessor is either < b or a 
right-to-left maximum (max for short). For example, the mid-123 entries in 13452 6 are 
3, 4, 5, 2 but only 3, 4, and 5 are key. 

Lemma 1. A permutation with no key mid-123 entries has no mid-123 entries at all and 
so is a 123-avoider. 

Proof. Suppose the ith entry 7Tj of a permutation n is a mid-123 entry but not key. 
Then, by definition of key, we have 7Tj_i > 7i~j and 7Tj_i is not a right-to-left max. So 
7Tj_i is also a mid-123 entry. Again, if 7Tj_i is not key, then 7Tj_ 2 > TTj-i and 7Tj_ 2 is n °t 
a right-to-left max. Iterating this process, we must eventually arrive at a key mid-123 
entry. Hence, every permutation with a mid-123 entry has a key mid-123 entry . □ 

Lemma 2. Suppose b is the last mid-123 entry in a {1243, 2134} -avoider. Then there is 
a unique c such that be forms the "23" of a 123 pattern. 

Proof. Suppose abc and abd are 123 patterns with c 7^ d, say c < d . If c precedes 
d in the permutation, then c is a mid-123, violating the hypothesis on b. If c follows d, 
then abdc is a proscribed 1243. □ 

Let A n denote the set of start-small {1243, 2134}-avoiders on [n], and A n ,k the subset 
with k key mid- 123 entries. 
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To produce the promised bijection <ft from A n to lists of start-small 123-avoiders whose 
total length is n — 1 + the length of the list, it suffices to exhibit, for < k < n — 2, a 
bijection 

A n ,k — > {(ci, ■ ■ , o"fc+i) : each <7j is a start-small 123-avoider, 

lengths of the cx^'s sum to n + k\. 

For = 0, naturally the bijection is it — > (it), a singleton list, because, by Lemma I , tt is 
already 123-avoiding. For 1 < k < n — 2 and k + 1 < j < n — 1 let 

•A n ,k,j = {vr G *4 n ,fc : the last mid-123 entry of 7r is in position j}. 

The result of the next Section extracts from a start-small {1243, 2134}-avoider that does 
contain 123's two {1243, 2134}-avoiders, both start-small, the first with one fewer key 
mid-123 entries than the original and the second with no key mid-123 entries. Iteration 
then gives the desired bijection <f>. 

3 The crucial bijection 

Proposition 3. For l<k<j<n — 1, there is a bijection 

•An,k,j ^ fc— 1 X -4 j, • 

Proof. Given n G An t k,j, we need to reversibly produce U\ G Aj t k-i an d cr2 G ^n+i-j,o- 
Write tt as r 1 6r 2 where b is the last mid-123 entry in tt, and let abc be the 123 pattern 
in tt with smallest a (c is uniquely determined by Lemma 2). Concatenate a and r 2 and 
standardize (replace smallest entry by 1, next smallest by 2, and so on) to get the desired 
o"2 with no 123's. 

Concatenate T\ and c to get a {1243, 2134}-avoider p — a candidate (after standard- 
ization) for 0i. This p may need further processing because of two glitches: p may still 
have k key mid-123's instead of the required k — 1 and p cannot end with its smallest 
entry (which must be possible in G\). But these glitches cancel out. 

If b is a key mid-123 in tt, then p has k — 1 key mid-123's because b has been lost. In 
this case, just standardize p to get a\. Otherwise, the last entry of T\ exceeds b but is not 
a right-to-left max. Delete from p the longest terminal string of T\ that is decreasing but 
does not contain a right-to- left max of tt (equivalently, does not contain an entry > c); 
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A {1243, 2134}-avolder, 
last mid- 123, b- 6, is key 
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A {1243, 2134}-avoider, 
last mid-123, h-5, is not key 



say r > 1 entries are deleted. Add r to each remaining entry of p, append the entries 
r, r — 1, . . . , 1 and standardize to obtain <j\. 

For example, for vr = 11 2 12 9 7 8 4 5 6 1 10 3, we find that the last mid-123, 6 = 6, 
is a key mid-123, and a = 2, c = 10. So cr 2 = standardize^ 1 10 3) = 2 1 4 3 and <j\ = 
standardize^ 2 12 97845 10) = 81964523 7. 

As another example, for vr = 13 16 12 3 15 8 9 10 11 7 6 5 2 1 14 4, we find that the last 
mid-123, b = 5, is not key, and a = 3, c = 14. So o~2 = standardize^ 2 1 14 4) = 3 2 1 5 4. 
Here, T\ = 13 16 12 3 15 8 9 10 11 7 6 and the longest decreasing terminal string of T\ 
that does not contain a right-to-left max of it has length r = 3. So ci = standardize^ + 
(13 16 12 3 15 8 9 10 11 7 6 14) 3 2 1) = standardize(16 19 15 6 18 11 12 13 17 3 2 1) = 
9 12 8 4 11 5 6 7 10 3 2 1. 

The invertibility of this map rests on structural properties of {1243, 2134}-avoiders 
evident in the matrix diagrams above: consider scanning the entries leftward from the 
last mid-123 entry b. In case b is key, as long as these entries decrease, they decrease by 
1 (in square box at center of yellow region). In case b is not key, they increase by 1 until 
either an entry > c, a decrease, or a jump > 1 to an entry < c occurs. In the latter case, 
the "missing" entries in the jump occur in increasing order immediately to the left of the 
jump (again in square box at center of yellow region). In both cases, the gray regions 
are empty except for the "a" and "c" , and the yellow square box is bisected by the main 
diagonal. 

The preceding observations, all immediate consequences of the patterns 1243 and 2134 
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being proscribed and b being the last mid-123, validate the following description of the 
inverse map. For a permutation a, we use a[j] to denote the entry in position j and a\j, k] 
to denote the list of entries occupying positions j through k. 

Suppose given a {1243, 2134}-avoider a x and a 123-avoider cx 2 and we wish to recapture 
7r. We have n = length^) + length(<7 2 ) — 1 and j = length(ai). Define the integer r 
by writing o\ in the form /i,r,r — 1,...,2, 1 with r maximal, where it is understood 
that fj, is <7i and r is if o\ does not end with 1. Set p = j — r, the length of fi, and 
q = length (<r 2 ) — 1. The positions i and k of the "a" and "c" respectively in tc are given 
by i = position in ji of its minimum entry, and k = j — 1 + position in <r 2 of its maximum 
entry. The values of "a" and "c" are recaptured as a = <r 2 [l] and c = <J\[p\ + q. Define s 
to be the length of the longest increasing terminal string in + l,p\. 

Now concatenate the strings 

a 1 [l,p-s]+q, (Ti[p-s+l,p-l] +q-l, n-j+r + s, a 1 [p+l,j} + q, a 2 [2,g+l], 
where the next to last string is empty when p = j, equivalently, when r = 0. 

Finally, to obtain n from the resulting string, overwrite, with a and c respectively, the 
entries in the positions that a and c should occupy, namely, positions i and k. 

For instance, the second example above yielded 

<— s — > ^— r — > 
o x = 9 12 8 4 11 5 6 7 10 3 2 1, a 2 = 3 2 1 5 4, 

and, to reverse the map, we find n = 16, j = 12, r = 3, p = 9, g = 4, s = 4, i = 4, A; = 
(j — 1) + 4 = 15, a = cr 2 [l] =3, c = o"i [9] + 4 = 14. The strings to be concatenated are 

(9 12 8 4 11) +4, (5 6 7) + 3, 11, (3 2 1) + 4, (2 1 5 4), 

yielding 

13 16 12 8 15 8 9 10 11 7 6 5 2 1 & 4 , 

with entries in positions i — 4 and k — 15 crossed out. Replace them with a = 3 and 
c = 14, respectively, to recover n. 

4 Putting it all together 

A k-list is a list of length k. Recall that if A is a class (species) of combinatorial structures 
with a n structures of size n (n > 1), and a compositional ^4-structure of size n is one 



5 



obtained by taking a composition (n 1; n 2 , . . . , Uk) of n and forming a fc-list of .A-structures 
of respective sizes n>i, n 2 , . . . , n k , then the counting sequence (&„)„>i for compositional A- 
structures has generating function B(x) := J2 n >i b n x n given by 



B(x) = ' 



l-A(xY 

where A(x) is the generating function for ^.-structures. This is known as the INVERT 
transform [5] but bearing the combinatorial interpretation in mind, it could as well be 
called the Compositional transform. We will apply it to the class of (nonempty) start-small 
123-avoiders with size measured as "length minus 1" (note that there is no start-small 
123-avoider of length 1). 



With C{x) := 1 ^ 4x denoting the generating function for the Catalan numbers C n , 
which are well known to count Dyck paths of n upsteps, we compute 

[x n ] C(x) 3 = # ordered triples (P, Q, R) of Dyck paths with a total of n upsteps 

= # Dyck paths of n + 2 upsteps that start UU ((P,Q,R) -»■ UUPDQDR) 
= # start-small 123-avoiders on [n + 2] (via Krattenthaler's bijection [6]) . 

Thus, 

[x n }xC(xf = [x^^C^xf = # start-small 123-avoiders on [n + 1] 

= # start-small 123-avoiders of size n , 



and, using the Compositional transform, we have for n > 1 
1 

1 — x C(x 



xTl ] = # lists of start-small 123-avoiders of total size n 



# fc-lists of start-small 123-avoiders of total length n + k . 



k=l 



Hence, for n > 2, 



x 

\x 



n-1 



"1 ^, xo = # fc-lists of start-small 123-avoiders of total length n — 1 + k 

1 - xC(x) 3 

= # start-small {1243, 2134}-avoiders of length n (via the bijection < 

from which it is immediate that the generating function for start-small {1243, 2134}- 
avoiders (with x marking length) is 

x 

G(x) = 1 + -— 3 - X . 

1 — xC(x) 6 
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Now let {u n ) n >Q = (1,1,2,6,...) and (i> n ) n >o = (1,0,1,4,...) denote the counting 
sequences for {1243, 2134}-avoiders and start-small {1243, 2134}-avoiders respectively. 
Clearly, v n = u n — u n -i for n > 1, (consider deletion of the first entry from a {1243, 2134}- 
avoider on [n] that starts n). So the generating functions F(x) = J2 n >o UnXn an< ^ 
G(x) = J2 n>0 VnX n are related by F(x) = G(x)/(1 — x). Thus 

F(x) = = 1+ i-*c(*) 3 ~ X 

y ' l-x 1-x 

which, after expansion, agrees with Kotesovec's formula. 

Losonczy [7] has counted permutations that avoid 3421, 4312 and 4321 or equivalently 
(by reversal) both of the patterns treated here and 1234. 
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